DOCUMENT RESUME 



ED 358 36A 



CE 063 966 



AUTHOR 
TITLE 

PUB DATE 
NOTE 



PUB TYPE 



EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



van Batenburg, Theo A*; Lokman, Ineke H* 
Effectiveness in Senior Secondary Vocational 
Education* 
Jan 91 

lip*; Paper presented at the Annual Meeting of the 
ICSEI (Cardiff, Wales, United Kingdom, January 
1991) • 

Speeches/Conference Papers (150) — Reports - 
Research/Technical (143) 

MFOl/PCOl Plus Postage* 

'^Academic Achievement; Business Education; Foreign 
Countries ; High Schools ; '^Institutional 
Characteristics ; ^Outcomes of Education; 
Postsecondary Education; School Holding Power; 
^'School Role; Technical Education; '^Vocational 
Education 
"Netherlands 



ABSTRACT 

Research was conducted in the Netherlands to 
determine what variables at the school level contribute to the 
differences in school effectiveness in the senior secondary 
vocational education system. Information was gathered through 
questionnaires answered by a representative sample of 57 of the 
country's 276 senior vocational schools and through questionnaires 
and intelligence test results from 4,889 students* Students were 
asked about their age, motivation, and preceding education. Analyses 
using a logistic approach showed that some school characteristics 
have a small effect on the degree to which students pass on to the 
next school year. These variables include pupil tutoring, the quality 
of the educational situatior, and school size. However, student 
motivation and intelligence play a larger role in school success than 
school characteristics. (KC) 
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abstract 

The subject of this paper is the effectiveness of the senior secondary vocational education, A 
representative of each school filled in a questionnaire about the educational arrangements at 
school. The pupils filled in a queslionnair as well and were tested on intelligence- They w6re 
asked about their age, motivation and preceding education. The (logistic) multi-level analyses 
show that some school characterisiic^ have a small effect on the degree in which pupils pass on to 
the next schoolyear. 



Introduction 

Research in senior secondary vocational education has not received much attention with respect 
to school effectiveness. This is rather strange because 42% (n =300.000) of the Dutch generation 
pupils take part in it. About 20% of these pupils are found in the senior secondary commercial 
section, 22% in the senior secondary technical section and 21% in the senior secondary personal 
and social services and health care section. The school effectiveness of these sections is the 
subject of this article. The pupils vary in age from 16 until about 20 years of age. To get an idea 
of the sort of professions these pupils are trained for some of the many obtainable professions 
vvall be summed up here. In the senior secondary commercial section pupils are trained to become 
a secretary, bookkeeper, salesman and so on; in the senior secondary technical section: electrician, 
clerk of works, draughtsman; in the senior secondary personal and social services and health care 
section: nurse, assistent to the doctor, cook in institutions. The senior secondary commercial 
section has a general first year. In the other two sections the pupils have already chosen for a 
specilization in a group of professions. 

In this article we will try to contribute to the theory of school effectiveness by exploring effective 
school variables for the specific field of senior f econdary vocational education. 
The output of the senior secondary vocational education seems to be rather low. In the senior 
secondary commercial section about 13% of the pupils fail their first year and 24% have already 
dropped out of school during the first year. For the senior secondary technical section these 
percentages are 12 and 14, and for the senior secondary personal and social services and health 
care section 6 and 19. The effectiveness of the senior secondary commercial section is very low, 
the other sections seem to be somewhat more effective but their percentages are still not very 
outstanding. The above mentioned percentages may vary considerably between schools. For failing 
the first year we found in the senior secondary commercial section percentages varying between 
7% and 27%, for drop out between 10% and 43%. In the senior secondary technical section these 
percentages vary for failing between 0% and 28%, for drop out between 6% and 29%. In the 
senior secondary personal and social services and health care section between 0% and 19% for 
failing and between 5% and 33% for drop out. 

These variations in percentages are an indication for rather large differences in school 
effectiveness. The following research question will be explored: "which are the variables at the 
school level that contribute to the differences in school effectiveness in the senior secondary 
vocational education"? 



Variables at the level of the school 

Vocational education has a specific character. In comparison to forms of general education the 
major emphasis is on the preparation for a profession and to a lesser degree on general cognitive 
objectives. This is the reason why we have taken passing or not passing the first year as the effect 
criterion. It is expected that aspects of presenting the subject matter in modules, learning in 
practice, aspects of pupil tutoring and the attitude of the school towards educational reform have 
some relation with the criterion. Also the school-size will be taken into consideration. 
These variables have been measured by means of a questionnaire filled in by the school 
administrator. The following concepts have been measured in item scales: history of educational 
reform in the periods 1982-1987 and 1987-1989; pupil tutoring; the educational process and the 
judgement about the quality of the educational situation. This item scales are reasonably reliable 
with Cronbachs' alpha's all above .65. For this article it takes too much space to present all items 
and the scale analyses. However to give an idea about the contents of the scales we present for 
each scale some examples of items. The two scales about the 'History of Educational Reform' 
(HER1&2) contain items like the following: which educational reform has been realized regarding 
"educational objectives", "presenting the subject-matter in modules", "the ways of learning in 
practice" and "examination". The scale named 'Pupil Tutoring' (PT) contains items like: " is there 
time available at school in which pupils with educational deficits get additional attention", "does 
social emotional tutoring exiot at school", " is there time available at school in which pupils can 
make their homework and ask a teacher questions". The scale named 'Educational Process'(EP) 
contains items like: it is customary that "pupils work in groups", "choose their own topics for 
papers". In the scale 'Quality of the Educational Situation' (QES) the school administrator made a 
judgement about the quality of the school equipment, the quality of the subject-matter and the 
financial position of the school. The school-size and whether or not the groups are formed on 
basis of previous education are taken in consideration as well (grouping versus no grouping), with 
the exception of the senior secondary personal and social services and health aire section where 
no grouping procedure exists. 

Variables al the level of the pupfl 

School effects cannot be estimated without taking pupil variables into consideration. The 
intelligence of pupils has been tested in two subtests of the PSB (Horn, 1969). This test is 
relatively easy to administer within a short period of time. These subtests concern the non-verbal 
part of intelligence. The motivation of the pupils has been measured by means of an item scale 
(alpha = .80). This scale contains 15 items like: "it is important for me to obtain good results at 
school", "I always make my homework", "I think of myself as a motivated pupil". The pupils were 



also asked about their previous education which can be 'Junior secondary vocational education* 
(JSVE), 'Junior general secondary education'(JGSE) and a remaining category of all kinds of 
courses (other). 

Data and method 

In the following table the data are presented. 
Table 1: the data. 

total number of 



section 


pupils 


schools 


schools 


senior secondary commercial education 


3434 


18 


75 


senior secondary technical 


777 


20 


69 


senior secondary personal and social 


678 


19 


132 



services and health care education 

Since the data are both on the level of the school and on the level of the pupil a multi-level 
model will be used to estimate the effects. This will be done with the computer program VARCL 
(Longford, 1986). In this multi-level model the logistic option (binomial distribution) has been 
chosen (Longford, 1988; Goldstein, 1987). This is a suitable way of treating dichotomous 
dependent variables. All analyses will be done seperately for each section. 

At first an empty model will be estimated to get an impression of the ,.mount of variance at the 
level of the pupil and at the level of the school. Secondly all variables at the level of the pupils 
will be modelled to find the relative importance of the variables. In the third place only significant 
pupil variables will be modelled. In the fourth place all variables at the school level will be 
modelled together with the significant pupil variables. In the fifth place only significant pupil and 
school variables will be modelled. In the sixth place the pupil variables within the schools will be 
considered as having random effects (i.e. effects varying over schools) to explore within-group 
heterogeneity. All these models will be evaluated with regard to their differences in model fit. 
The three sections of vocational education have their own character, their curriculum is 
completely different. That is the reason why we will model the variables within the sections. An 
other possible approach is an overall modelling procedure in which the section is a dummy 
variable. This procedure is less valid and more difficult to interpret because all interaction terms 
with the dummy variable must be explained. In this procedure effects within small sections are 
overruled by a Aon effect in a larger section. 

Before we started the modelling procedure all variables have been transformed into z-scores 
within sectionfi. This was done in order to be able to compare effect sizes. A larger value 



indicates a larger effect size. However, in the binomial distribution these effect sizes cannot be 
directly transformed into the probability of passing or not passing the year. These probabilities 
depend on the baseline value (the value for a zero effect size) of the probability function. The 
same effect size will add more to the probability when the value is .5 than when it is .9. In 
diagram 1 this will be made clear. 

Oiagrun 1: effect size and probability of passing the year 
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Diagram 1 shows the case where there are equal effect sizes and different probabilities of passing 
on to the next year differs (a=b;c>d). 



Results 

As was mentioned before a stepwise modelling procedure has been be used: 

- model 1 : the empty model 

- model 2 : all variables at the level of the pupil are included 

- model 3 : only significant pupil variables are modelled 

- model 4 : all school variables are included in model 3, together with the significant pupil 

variables 

- model 5 : only significant school and pupil variables will be included. This model will be 

presented only if significant effect of school variables are found. 

- model 6 : the significant pupil variables will be modelled as variables with random effects in 

order to explore within-group heterogeneity. This last model will be presented only if 
such random effects are found. 

In table 2 the results of the modelling procedure in the senior secondary commercial section are 
presented. 



Table 2: models in senior secondary commercial education for passing or not passing the first 
year 



modell model2 modeB model4 



r.i.v. .15 1 


.11 


.11 


.07 


deviance 4454 } 
PSB3 


4339 


4340 


4339 


.11* 


.13* 


.13* 




(.04)# 


(.04) 


(.04) 


PSB8 


.04 








(.04) 






AGE 


-.24* 


-.24* 


-.24* 




(.04) 


(.04) 


(.04) 


MOTIVATION 


.15* 


.15* 


.15* 




(.04) 


(.04) 


(.04) 


-JSVE 


.00 


.00 


.00 




(.00) 


(.00) 


(.00) 


-JGSE 


.67* 


.66* 


.67* 




(.11) 


(.11) 


(.11) 


-other 


.96* 


.96* 


.97* 




(.20) 


(.19) 


(.19) 


HERl 






.33 








(.20) 


HER2 






-.38 








(.20) 


FT 






-.11 








(.08) 


EP 






.03 








(.09) 


QES 






-.03 








(.09) 


school-size 






.01 








(.09) 


grouping 






.00 








(.00) 


no-grouping 






.01 








(.18) 



r.i.v. =random intercept variance 
* =signi£icant effect 

#= standard error always between brackets 



The grand mean on the level of the pupil equals 1.00. The variance to explain at the school level 
is calculated as follows: the random intercept variance * the probability for passing the year * ( 1 - 
the probability for passing the year) * 100%. The mean probability for passing the year is .63 in 
the senior secondary commercial section, so the variance to explain at the school level equals .15 
* .63 * .37 * 100% = 3.4%. When the pupil variables are entered this percentage drops to 2.5,%. 
Model 2 is a significant improvement to model 1. The difference in deviances is 115 with 6 
degrees of freedom (chi squared = 12.6). Model 3 is the best model in wich only significant effects 
at the level of the pupils are included. The difference between the deviances of model 2 and 



model 3 is not significant. The intelligence of the pupils (PSB3) and motivation has a small effect. 
It seems clear that older pupils more often fail the first year. Pupils with a previous junior 
secondary vocational education (JSVE) fail more often than pupils with a junior general 
secondary education (JGSE) and other previous education. The school variables in model 4 do 
not explain any of the variance. 

In table 3 the results of the modelling procedure in the senior secondary technical section are 
presented. 



Table 3: models in senior technical secondary vocational education. 





modell 


model2 


model3 


model'i 


model5 


r.i.v. 


.23 


.25 


.24 


.01 


.04 


deviance 


949 


937 


946 


925 


929 


PSB3 




.08 

(.09)# 








PSB8 




.11 
(.09) 








AGE 




-.12 
(.08) 








MOTIVATION 




.17* 


.15 


.16* 


.15 






(.08) 


(.08) 


(.08) 


(.08) 


-JoVJt 




.UU 

(.00) 












1 '7 

(.18) 








-other 




(.37) 








HERl 








.07 
(.16) 




HER2 








-.02 
(.20) 




PT 








.37* 
(.16) 


.38* 
(.10) 


-P 








-.20 
(.14) 




QES 








.27* 
(.13) 


.19 
(.11) 


school-size 








-.36* 
(.15) 


-.37* 
(.10) 


grouping 








.00 
(.00) 




no-grouping 








-.31 
(.31) 





r.i.v. =random intercept variance 
*=significant effect 

#= standard error always between brackets 
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The empty model 1 shows us that the random intercept variance is .23 which means that by a 
mean probability for passing the year of .65, 5.2% of the variance might be explained by variables 
at the school level. When the pupil variables are entered in model 2 this percentage rises to 5.7%. 
However the difference between the deviances of model 1 and model 2 is not significant (chi 
squared =12.6 df=6). Only the motivation of pupils seems to have a slightly positive impact. 
When in model 4 the school variables are entered the random intercept variance drops to ,01 
wich means that these variables explain a large part of the variance between schools. The deviance 
of model 5 differs significantly of the deviance in model 1. Pupil tutoring (PT) and the quality of 
the educational situation (QES) have a positive effect. School-size has a negative effect, on larger 
senior secondary technical schools the pupils pass less often to the second year. 

In table 4 the results of the modelling procedure in the senior secondary personal and social 
services and health care section are presented. 

Table 4: models in senior secondary personal and social services and health care education. 



model 1 model2 model3 model4 



r i.v. .07 


.03 


.04 


.02 


deviance 762 


717 


718 


711 


PSB3 


.32* 


.28* 


.27* 




(.10)# 


(.09) 


(.09) 


PSB8 


-.10 








(.10) 






AGE 


-.01 








(.09) 






MOTIVATION 


.52* 


.52* 


.51* 




(.10) 


(.10) 


(.10) 


-JSVE 


.00 


.00 


.00 




(.00) 


(.00) 


(.00) 


-JGSE 


.54* 


.54* 


.50* 




(.20) 


(.20) 


(.20) 


-other 


.59 


.57 


.53 




f.33) 


(.32) 


(-32) 


HERl 






-.29* 








(.14) 


PT 






.00 








(.12) 


EP 






-.21 








(.14) 


school-size 






.19 








(.13) 



r.i.v.= random intercept variance 
*=significant effect 

#= standard error always between brackets 



0 



In the senior secondary personal and social services and health care section the mean probability 
for passing the year equals .75. The random intercept variance of model 1 is .07 which implies 
that only 1.3% of the variance can be explained with variables at the school level. When in model 
2 the pupil variables are included the random intercept variance drops to .03. This indicates that 
the variables at the school level can not have a large effect. Model 4 is a significant improvement 
of the empty model (chi squared =12.6, df=6). The best model is model 3 because omitting non 
significant effects does not decrease the deviance much. Intelligent and motivated pupils with a 
junior general secondary education pass more often to the next year. 

Discussion 

By means of the logistic option of the VARCL-program we have tried to estimate effects of 
variables at ihe school level on passing or not passing the first year in the senior secondary 
vocational education. In the analysed models also effects of variables at the level of the pupil have 
been taken into consideration. 

In the senior secondary commercial section and the senior secondary personal and social services 
and health care section there is not much variance to explain at the school level (3.4% and 1.3% 
respectively). In the senior secondary commercial section the school variables do not explain any 
variance at the school level, while in the senior secondary personal and social services and health 
care section only the history of educational reform has a small negative effect. In these sections 
mainly pupil variables like intelligence, age, previous education and motivation could explain 
whether or not pupils pass on to the second school-year. 

In the senior secondary technical section the variance to explain at the school level is the highest 
(5.2%). Most of the variance could be explained by the school variables. Effects are found for 
pupil tutoring, the quality of the educational situation and school-size. At the pupil level only 
motivation has a small effect. 

We can conclude that the amount of variance to explain at the school level is very small and that 
the effects of variables differ between sections. The small amount of variance to explain at the 
school level could result from the dichotomous dependent variable. The probability for passing the 
year varies between .63 and .75, while the amount of variance is maximal by a probability of .50 
for passing the year (the variance of the probability for passing the year = the probability for 
passing * (1 - the probability for passing)). 

Probably the differences between sections can be attribute to their selection policy. In the senior 
seconary technical section pupils are more intelligent and motivated than the pupils in the other 
sections. This made us believe that if pupils' intelligence and motivation are above a certain limit 
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these variables become less important and school variables become more important to explain 
whether or not pupils pass on. 
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